Surgical system with AR/VR training simulator and intra-operative physician image-guided assistance

ABSTRACT

Technologies for providing assistance to a surgeon during a surgical procedure are disclosed. An example method includes identifying a medical condition of a patient and recommending surgical procedures for treatment. An optimal surgical procedure is then determined based on correlations between the medical condition of the patient and outcomes of previous surgical procedures for other patients previously suffering from the medical condition. A 3D model of the patient may be created using current images of an affected area of the patient. Successively, a surgeon is trained using a Virtual Reality simulation. During the training, the surgeon may be allowed to provide annotations. Further, the recommended surgical procedures are based on medical data of the patient, including medical images. Surgical paths are retrieved for addressing the surgical need of the patient, and the surgical paths may be overlaid on image segments, for display for the surgeon. A surgical path is selected from the surgical paths, a surgical step belonging to the surgical path is selected, and an image segment selected from the image segments, based on the surgeon&#39;s inputs is used to create the surgical plan.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/012,464, filed on Jun. 19, 2018 (now U.S. Pat. No. 11,350,994), whichclaims the benefit of U.S. Provisional Application No. 62/521,537, filedon Jun. 19, 2017, and U.S. Provisional Application No. 62/528,480, filedon Jul. 4, 2017, all of which are incorporated herein by reference intheir entireties.

TECHNICAL FIELD

Technologies described herein generally relate to users performingsurgical simulations using an AR/VR training simulator andintra-operative image-guided assistance.=.

BACKGROUND

The subject matter discussed in the background section should not beassumed to be prior art merely as a result of its mention in thebackground section. Similarly, a problem mentioned in the backgroundsection or associated with the subject matter of the background sectionshould not be assumed to have been previously recognized in the priorart. The subject matter in the background section merely representsdifferent approaches, which in and of themselves may also correspond toimplementations of the claimed technology.

Each surgical procedure has a different level of associated risk. Therisk may arise due to the medical health of the patient before surgery,as well as any errors or shortcomings during surgery. To minimize therisks arising due to a patient's health, a pre-operative evaluation isgenerally performed. The pre-operative evaluation is performed toidentify a patient's history of disease, allergies, and current medicalcondition. The pre-operative evaluation may include a physicalexamination, neurological examination, and/or other specialized medicaltests. The pre-operative evaluation enables a medical staff to takepro-active steps to reduce risks associated with a forthcoming surgicalprocedure.

Based on the pre-operative evaluation, a surgeon defines hispre-operative plans and steps for executing a surgical procedure. Thepre-operative plans may include a patient's dosage of medication before,during, and after the surgical procedure; surgical devices to be usedduring the procedure; and a method of handling the surgical devicesduring the surgical procedure.

A surgeon may also have in mind several machines and methods foroperating efficiently on a patient by improving surgical accuracy andminimizing risks during the surgical procedure. Such methods may includestoring images of an affected or target area of the patient, storingvideos of similar surgical procedures for reference, and training onartificial organs.

During an image-guided surgery, a surgeon makes use of tracked surgicalinstruments in conjunction with images of a patient, i.e., the surgeonuses images of an ultrasound in order to guide needles being insertedinto a liver of a patient for ablation.

Currently, image-guided surgery provides various techniques, such asThree-Dimensional (3D) graphics rendering techniques, medical imageshape detection, etc. However, current techniques are prone toinadequate image management. For example, the current techniques requirethe surgeon to forego performing other activities during the surgicalprocedure in order to manipulate a view of Computerized Tomography (CT)scans, Magnetic Resonance Imaging (MRI) images, and other visualizationmedical data, using a mouse-and-monitor interface. Such type ofsuspension or disruption of the surgical procedure may result inincreased time and reduced fluidity.

Further, current techniques for image-guided surgery do not provide asufficient control and flexibility in viewing the CT scans and the othervisualization medical data due to unintuitive and difficult interfaces.For example, 3D viewing systems may display obscured areas of thevisualization medical data. Also, the current techniques fail to providean adequate method to the surgeon for annotating the images during thesurgical procedure. But such techniques are difficult, time-consuming,and disruptive.

Further, robotic surgery is a branch of science by which a surgeon usesrobots for operating on patients. Such robots improve precision of asurgeon by providing greater control during each stage of a surgicalprocedure, and thereby reduce chances of errors. Further, severalsystems provide the surgeons with remote control of robotic arms presentin a remote operation theatre by viewing a real-time video display of asurgical area. Based on the video of the patient, the surgeon mayremotely control the robotic arms to perform the surgery. Usage of suchrobots allows the surgeons to minimize entry wounds, reduce blood lossduring the surgery, and leave least amount of scarring after thesurgery. For example, the Da Vinci® robotic surgical system developed byImperial College London has been used in general surgical proceduressince the early 2000s.

Artificial Intelligence (AI) technologies including machine vision andimage analysis complement surgeons' abilities. A few of the alreadyidentified uses include highlighting blood vessels, nerve cells, tumormargins, and other important structures that are hard to recognize bynaked eye or extrapolation on a screen. Thus, AI may improve a surgeon'sanalysis of real time visual data of patients, thus improving theirsurgical procedure.

Augmented Reality (AR) systems are also used by surgeons for overlayingof important information required during a surgical procedure. Suchinformation includes pre-operative images, lab test results, and detailsof previous such surgical procedure. The information is displayed onmultiple monitors stacked around a surgeon in an operation theatre. Theinformation is used by the surgeon while performing the surgicalprocedure.

Thus, the current state of art merely involves usage of images, labresults, and other historical data related to a patient. All such datais simply projected on a screen for a surgeon's reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate various embodiments of systems,methods, and embodiments of various other aspects of the disclosure. Anyperson with ordinary skills in the art will appreciate that theillustrated element boundaries (e.g., boxes, groups of boxes, or othershapes) in the figures represent one example of the boundaries. It maybe that in some examples one element may be designed as multipleelements or that multiple elements may be designed as one element. Insome examples, an element shown as an internal component of one elementmay be implemented as an external component in another, and vice versa.Furthermore, elements may not be drawn to scale. Non-limiting andnon-exhaustive descriptions are described with reference to thefollowing drawings. The components in the figures are not necessarily toscale, emphasis instead being placed upon illustrating principles.

FIG. 1A illustrates a network connection diagram 100 of a system 102 forproviding surgical assistance during a surgical procedure, according toan embodiment.

FIG. 1B illustrates a network connection diagram 1000 of a system 102for creating a surgical plan for use in a surgical procedure, accordingto another embodiment.

FIG. 2A shows a block diagram showing different components of the system102, according to an embodiment.

FIG. 2B shows a block diagram showing different components of the system102, according to an embodiment.

FIGS. 3A and 3B collectively illustrate a flowchart 300 showing a methodexecuted by procedure planning module 210 of the system 102, accordingto an embodiment.

FIG. 4 illustrates a Graphical User Interface (GUI) of a user device120, presenting information related to surgical procedures, according toan embodiment.

FIG. 5 illustrates a flowchart 500 showing a method executed by VirtualReality (VR) surgical practice module 212 of the system 102, accordingto an embodiment.

FIG. 6 illustrates a flowchart 600 showing a method executed byAugmented Reality (AR) assistance module 214 of the system 102,according to an embodiment.

FIG. 7 illustrates a method 700 of providing surgical assistance duringa surgical procedure, according to an embodiment.

FIG. 8 illustrates a flowchart 300 showing a method performed by asurgical planning module 210 of the system 102, according to anembodiment.

FIG. 9 illustrates a flowchart 400 showing a method performed by asurgical annotation module 212 of the system 102, according to anembodiment.

FIGS. 10A-10F collectively illustrate a Graphical User Interface (GUI)500 of a user device 116 connected to the system 102, presentinginformation related to the surgical procedure, according to anembodiment.

FIG. 11 illustrates a flowchart 600 showing a method for creating thesurgical plan for use in the surgical procedure, according to anembodiment.

DETAILED DESCRIPTION

Some embodiments of this disclosure, illustrating all its features, willnow be discussed in detail. The words “comprising,” “having,”“containing,” and “including,” and other forms thereof, are intended tobe equivalent in meaning and be open ended in that an item or itemsfollowing any one of these words is not meant to be an exhaustivelisting of such item or items, or meant to be limited to only the listeditem or items.

It must also be noted that as used herein and in the appended claims,the singular forms “a,” “an,” and “the” include plural references unlessthe context clearly dictates otherwise. Although any systems and methodssimilar or equivalent to those described herein can be used in thepractice or testing of embodiments of the present disclosure, thepreferred, systems and methods are now described.

Embodiments of the present disclosure will be described more fullyhereinafter with reference to the accompanying drawings in which likenumerals represent like elements throughout the several figures, and inwhich example embodiments are shown. Embodiments of the claims may,however, be embodied in many different forms and should not be construedas limited to the embodiments set forth herein. The examples set forthherein are non-limiting examples and are merely examples among otherpossible examples. It will be readily understood that the aspects of thepresent disclosure, as generally described herein, and illustrated inthe figures, can be arranged, substituted, combined, separated, anddesigned in a wide variety of different configurations, all of which areexplicitly contemplated herein.

FIG. 1A illustrates network connection diagram 100 of a system 102 forproviding surgical assistance during a surgical procedure, according toan embodiment. The system 102 may be connected to a communicationnetwork 104. The communication network 104 may further be connected witha healthcare facility 106 for facilitating data transfer between thesystem 102 and the healthcare facility 106.

The communication network 104 may be a wired and/or a wireless network.The communication network 104, if wireless, may be implemented usingcommunication techniques such as Visible Light Communication (VLC),Worldwide Interoperability for Microwave Access (WiMAX), Long TermEvolution (LTE), Wireless Local Area Network (WLAN), Infrared (IR)communication, Public Switched Telephone Network (PSTN), Radio waves,and other communication techniques known in the art.

The healthcare facility 106 may comprise a real-time health record unit108 and a group of databases for storing different information requiredduring a surgical procedure. The group of databases may comprise asurgical outcomes database 110, surgical options database 112, surgicalannotations database 114, and a correlation database 116. Differentdatabases are used in present case; however, a single database may alsobe used for storing the data. Usage of the different databases may alsoallow segregated storage of different data and may thus reduce time toaccess required data.

In one embodiment, the surgical outcomes database 110 may be configuredto store results of previous surgeries for several patients. The resultsmay be stored in a structured manner. In one case, the results may beorganized based on types of surgical procedures, followed by patients'medical histories, followed by pre-plan of surgical procedures, andbefore and after annotations of surgeons for the surgical procedures.

The surgical outcomes database 110 may further store images of thepatients. The images may be any of camera images, Magnetic ResonanceImaging (MRI) images, and X-Ray images. The surgical outcomes database110 may also comprise actual VR training data used by a surgeon, forpre-planning of a surgical procedure. The surgical outcomes database 110may also include data by a Three-Dimensional (3D) camera during theactual surgical procedure and all related monitor data, which isdisplayed on a screen during the surgical procedure. The surgicaloutcomes database 110 may also include unexpected or adverse eventsoccurring in a time-sequence of pre-plan and actual results. Thesurgical outcomes database 110 may also include annotations provided byprevious surgeons.

In at least one embodiment, the surgical options database 112 may beimplemented as a query on the surgical outcomes database 110. Thesurgical options database 112 may comprise data organized in a surgicaloptions structure. In at least one scenario facilitated by theembodiment, for a particular surgical procedure performed on aparticular type of patient, the surgical options database 112 may storeinformation related to each event step, such as details of the eventsthat went well, occurrence of adverse events, frequency of the adverseevents, annotations made by a surgeon, and usefulness of theannotations.

In at least one embodiment, the surgical annotations database 114 may beimplemented as a query on the surgical outcomes database 110. Thesurgical annotations database 114 may extract a surgeon's annotationsfrom the surgical outcomes database 110 and may organize the surgeon'sannotations. The surgical annotations database 114 may also input acurrent surgeon's annotations, made either during training andpreplanning or during an actual surgical procedure by the currentsurgeon.

In at least one embodiment, the correlation database 116 may beimplemented as a real-time query of all databases (surgical outcomesdatabase 110, surgical options database 112, and surgical annotationsdatabase 114) to find correlations based upon a current surgicalprocedure. For example, an adverse event such as a patient's bloodpressure unexpectedly dropping may occur during a surgical procedure.Accordingly, the correlation database 116 may implement a search of alldatabases for data relating to unexpected drop in blood pressure duringa similar surgical procedure. In accordance with this non-limitingexample, the data stored in the correlation database 116 may includedata indicating the percentage of patients who incur an increase inblood pressure during such a surgical procedure, as well as datacorrelating such occurrence of increased blood pressure to therespective patient's medical history. For example, the past history mayindicate that more than 75% patients experiencing increased bloodpressure were diabetic patients and that a particular injection (e.g.,Hydralazine used to relax and dilate blood vessels, resulting in loweredblood pressure) proved to be effective for 90% of such patients. Thecorrelation database 116 may also include a surgeon's annotationsorganized by frequency of usage of the database. For example, 70% of thesurgeons may have annotated to provide said injection before proceedingto a next step.

FIG. 1B illustrates a network connection diagram 1000 of a system 1102for creating a surgical plan for use in a surgical procedure, accordingto another embodiment. The system 1102 may be connected to acommunication network 104, similar to that of FIG. 1A. The communicationnetwork 104 may further be connected with the healthcare facility 106for facilitating data transfer between the system 1102 and thehealthcare facility 106.

The communication network 104 may be the same or similar to that of FIG.1A.

The healthcare facility 106 may include a real-time health record unit108 and a group of databases for storing different information requiredduring the surgical procedure. The group of databases may include asurgical annotation database 1110, an image abnormality database 1112,and a surgical path database 1114. As with FIG. 1A, different databasesare used in present case; however, a single database may also be usedfor storing the data. Usage of the different databases may also allowsegregated storage of different data and may thus reduce time to accessrequired data.

The real-time health record unit 108 may be configured to store medicaldata for a plurality of patients in real-time. The data may correspondto a respective patient's medical imaging data, diagnostic data, as wellas comprehensive medical history data. The stored medical data mayinclude medical images of an affected body part of a respective patient,as well as comprehensive medical records, including medical history,previous test results, and notes of surgeons/doctors or medicaladvisors.

In at least one embodiment, the surgical annotation database 1110 may beconfigured to accept annotations provided by a respective surgeon,either during surgical planning or in real-time during a surgicalprocedure. The surgeon may add the annotations at any time duringsurgical pre-operative planning, and the annotations may be stored inthe surgical annotation database 1110. The surgeon may add theannotations either using the system 102 or a user device 116. A smartphone is shown as the user device 1116 in FIG. 1B, as an example, usedfor displaying information related to the surgical procedures. However,user device 1116 may be implemented as any other device comprising aGraphical User Interface (GUI), for example, a laptop, a desktop, atablet, a phablet, or other such devices known in the art.

In at least one embodiment, the image abnormality database 1112 may beconfigured to store data related to irregularities and/or abnormalitiesin the medical images of the affected body part of a respective patient.The medical images may include a plurality of images captured using animaging system 118, utilizing known medical imaging techniques, such asMagnetic Resonance Imaging (MRI), Computerized Tomography (CT), X-rays,etc. The image abnormality database 1112 may store data related to theabnormalities identified in the medical images during the pre-planningstage of a surgical procedure. A non-limiting example of such anirregularity and/or abnormality stored in image abnormality database1112 may relate to a ruptured Achilles tendon of a right ankle of arespective patient.

In at least one embodiment, the surgical path database 1114 may beconfigured to store the plurality of surgical paths that may be followedfor respective surgical procedures.

FIG. 2A shows a block diagram showing different components of system 102as depicted in FIG. 1A. The system 102 comprises a processor 202,interface(s) 204, and memory 206. The processor 202 may execute analgorithm stored in the memory 206 for utilizing artificial intelligenceto provide surgical assistance. The processor 202 may also be configuredto decode and execute any instructions received from one or more otherelectronic devices or server(s). The processor 202 may include one ormore general purpose processors (e.g., INTEL® or Advanced Micro Devices®(AMD) microprocessors) and/or one or more special purpose processors(e.g., digital signal processors or Xilinx® System On Chip (SOC) FieldProgrammable Gate Array (FPGA) processor). The processor 202 may beconfigured to execute one or more computer-readable programinstructions, such as program instructions to carry out any of thefunctions described in this description.

The interface(s) 204 may help a user to interact with the system 102.The interface(s) 204 of the system 102 may either accept an input fromthe user or provide an output to the user, or may perform both theactions. The interface(s) 204 may either be a Command Line Interface(CLI), Graphical User Interface (GUI), or a voice interface.

The memory 206 may include, but is not limited to, fixed (hard) drives,magnetic tape, floppy diskettes, optical disks, Compact Disc Read-OnlyMemories (CD-ROMs), and magneto-optical disks, semiconductor memories,such as ROMs, Random Access Memories (RAMs), Programmable Read-OnlyMemories (PROMs), Erasable PROMs (EPROMs), Electrically Erasable PROMs(EEPROMs), flash memory, magnetic or optical cards, or other type ofmedia/machine-readable medium suitable for storing electronicinstructions.

The memory 206 may comprise an Artificial Intelligence (AI) surgicalassistant 208, implemented as a program. The AI surgical assistant 208may comprise three modules i.e., procedure planning module 210, virtualreality surgical practice module 212, and augmented reality assistancemodule 214.

FIG. 2B shows a block diagram showing different components of the system1102, as depicted in FIG. 1B, according to an embodiment. The system1102 includes a processor 202, interface(s) 204, and a memory 206,similar to that of FIG. 2A. Further to the description thereofpertaining to FIG. 2A, the processor 202 may execute an algorithm storedin the memory 206 for creating the surgical plan for use in the surgicalprocedure. The processor 202 may also be configured to decode andexecute any instructions received from one or more other electronicdevices or server(s). Again, the processor 202 may include one or moregeneral purpose processors (e.g., INTEL® or Advanced Micro Devices®(AMD) microprocessors) and/or one or more special purpose processors(e.g., digital signal processors or Xilinx® System On Chip (SOC) FieldProgrammable Gate Array (FPGA) processor). The processor 202 may beconfigured to execute one or more computer-readable programinstructions, such as program instructions to carry out any of thefunctions described in this description.

Further to the description thereof pertaining to FIG. 2A, theinterface(s) 204 may facilitate a surgeon's interaction with the system102. The interface(s) 204 of the system 102 may accept input from thesurgeon and provide an output to the surgeon. The interface(s) 204 mayeither be a Command Line Interface (CLI), Graphical User Interface(GUI), or a voice interface.

Again, similar to the description thereof pertaining to FIG. 2A, thememory 206 may include, but is not limited to, fixed (hard) drives,magnetic tape, floppy diskettes, optical disks, Compact Disc Read-OnlyMemories (CD-ROMs), and magneto-optical disks, semiconductor memories,such as ROMs, Random Access Memories (RAMs), Programmable Read-OnlyMemories (PROMs), Erasable PROMs (EPROMs), Electrically Erasable PROMs(EEPROMs), flash memory, magnetic or optical cards, or other type ofmedia/machine-readable medium suitable for storing electronicinstructions.

In one embodiment, the memory 206 may include a pre-operative planningimage processing module 1208. The pre-operative planning imageprocessing module 1208 includes a surgical planning module 1210.Further, the surgical planning module 1210 includes a surgicalannotation module 212.

Functioning of the procedure planning module 210, as shown in FIG. 2A,will now be explained with reference to flowchart 300 shown in FIGS. 3Aand 3B. One skilled in the art will appreciate that, for this and otherprocesses and methods disclosed herein, the functions performed in theprocesses and methods may be implemented in differing order.Furthermore, the outlined steps and operations are only provided asexamples, and some of the steps and operations may be optional, combinedinto fewer steps and operations, or expanded into additional steps andoperations without detracting from the essence of the disclosedembodiments.

A user may log into the procedure planning module 210. The term “user”may refer to a surgeon, and may henceforth be used interchangeably inthe description. Optionally, the logging in procedure may be accompaniedwith authentication of the user. Upon logging in, the procedure planningmodule 210 may allow the user to identify a subject patient and receiveidentification information for the subject patient, at step 302. Thesubject patient may refer to a patient upon whom the surgeon willoperate. The procedure planning module 210 may be utilized to retrieverecords pertaining to the subject patient from Electronic Health Records(EHR) stored in real-time health record unit 108.

Thus, step 304 includes retrieving records of the subject patient, andidentifying the surgery to be performed for the subject patient. As anexample of a surgery that may be performed, the subject patient may needan Anterior Cruciate Ligament (ACL) repair of his left knee. Allpossible surgical procedures for repairing of the knee may thus beretrieved from the surgical options database 112. Further to theexample, the surgical procedure for repairing the knee may include acadaver graft, patellar tendon graft, and different methods ofperforming such surgical procedures. The surgical procedures may beperformed though arthroscopy, open, and/or other different surgicalpaths of going in on one side of the knee and another side of the knee.Also retrieved from surgical options database 112 at step 306, alongwith the possible surgical procedures, are attributes of other patientsoperated upon using such surgical procedures as well as details ofadverse events that have occurred or may occur during such surgicalprocedures.

At step 308, the procedure planning module 210 may createThree-dimensional (3D) models of both the subject patient's affectedarea as well as that for the other patients whose comparative data wasretrieved. At step 310, the 3D model of the subject patient may becompared to the 3D models of the other patients to determine similarityscores. At step 312, data belonging to the other patients may befiltered based on a threshold value of the determined similarity scores.As a non-limiting example, the threshold similarity score value may beset as 25%, thus, data belonging to the other patients, present in thetop 25% of the similarity scores may be kept for further analysis.

Step 314 includes determining best fit curves for attributes of theremaining ones of the comparative patients. For example, Body Mass Index(BMI) may be used as an attribute of the remaining ones of thecomparative patients. Step 314 may also include determining correlationcoefficients for experiencing adverse events from a surgical procedure.For example, blood loss during surgery may be one such adverse event.Thereafter, step 316 includes storing best fit curves for each attributein the correlation database 116.

Step 318 includes filtering the attributes based on a threshold value ofcorrelation coefficient. As a non-limiting example, the attributeshaving correlation coefficient values greater than the threshold valueof correlation coefficient may be filtered for further use. Thus, if thethreshold value of correlation coefficient is set at 0.6, the attributeshaving correlation coefficient values less than 0.6 may be discarded.Step 318 also includes identifying relevant adverse events based on thefiltering. The attributes in which the subject patient's attributemeasurement lies outside of a standard deviation from a best fit curvemay also be discarded.

Step 320 includes scoring a severity of the relevant adverse event maybe scored. As a non-limiting example, the score may range from 1 to 10,by which 1 may indicate a lowest level of severity and 10 may indicate ahighest level of severity. The severity scores for adverse events may bedetermined based on predefined rules, as severity of any adverse eventis well known to the surgeons.

At step 322, adverse event scores may be determined for each surgicalprocedure previously identified at step 306 by multiplying a correlationcoefficient of an attribute, present in common with the subject patient,by a severity score of the adverse event. For example, the correlationcoefficient between BMI greater than 35 and bleeding during patellartendon graft may be 0.8, and the severity score may have a value of 3.The severity of the adverse event during such case may be scored as“0.8*3=2.4.” In accordance with a non-limiting example, the score of 2.4may be summed with scores of all other adverse events. It is evidentthat all such scores are pre-determined for adverse events, and thus theadverse events occurring in a real surgical procedure may have corollaryscores.

FIG. 4 shows a non-limiting example of information related to thesurgical procedures being displayed on a Graphical User Interface (GUI)of a user device 120 such as the smart phone shown as the user device120 in FIG. 1 . However, the user device 120 used to display informationrelated to the surgical procedures may be any other device comprising aGUI, for example, a laptop, a desktop, a tablet, a phablet, or othersuch devices known in the art.

Step 324 of flowchart 300 continues, with reference to FIG. 4 , byranking the surgical procedures based on the respective adverse eventscores. The surgical procedures may be displayed on the GUI with theadverse events being listed from top to bottom in ascending order ofadverse event scores. A user may then select one surgical procedure fromamongst the available surgical procedures, and step 326 includes a userselection of the surgical procedure being accepted through the GUI.

In one embodiment, using the GUI, the user may view plurality ofinformation in real-time, during the surgical procedure. The pluralityof information may include the possible surgical procedures, currentlyadopted surgical procedure, adverse events ranked on the basis of theirscores, annotations related to the adverse events, associated fileshistory for an adverse event (for example, Patellar chip in presentcase), procedure to be followed at a current step to move ahead, and areal-time video of each step performed. Further, all images displayed onthe GUI may be overlaid, scaled, and annotated by the user. Thus, step328 includes sending to Virtual Reality (VR) surgical practice module212 the data related to the surgical procedure and adverse eventsrelated to the surgical procedure.

The VR surgical practice module 212 may allow a user to work on VRsurgical practice equipment 122, which may be connected to the system102, and may allow simulation of surgical procedures. The simulation mayinclude displaying one or more virtual organs to be operated upon by theuser. A vital organ may comprise multiple elements and each element mayhave neighbouring elements. A plurality of tensioned connections mayconnect the neighbouring elements with the vital organ, such that forceapplied on one element propagates via respective neighbouring elementsand thus providing a distributed reaction over the vital organ.

The VR surgical practice equipment 122 may also include a physicalmanipulation device for manipulation by the user and a trackingarrangement for tracking the physical manipulation device andtranslating motion of the physical manipulation device into applicationof forces onto said virtual organ. The VR surgical practice equipment122 facilitates simulation of moving, cutting, suturing, coagulations,and other aspects of surgical procedures for different organs. Thus, theVR surgical practice equipment 122 may facilitate surgical practice fora user.

FIG. 5 illustrates a flowchart 500 for a method executed by VR surgicalpractice module 212, as shown in FIG. 2A. A user may log into the system102 using VR surgical practice equipment 122. At step 502, the VRsurgical practice module 212 may receive data related to the surgicalprocedure and adverse events related to the surgical procedure. The usermay start VR practice session using the augmented reality display 118and the VR surgical practice equipment 122, based on the data receivedfrom the procedure planning module 210.

Step 504, during the VR practice session, includes the data pertainingto possible adverse events being displayed to the user in synchronismwith the same moments during the practice session at which the adverseevents actually occurred in previous surgical procedures. At step 506,the user may provide annotations related to different steps of surgicalprocedure training. The annotations may be made in various formatsincluding, but not limited to, text, audio notes, instructions to pullup specific medical data from patient's Electronic Health Records (EHR),and Audio-Video files related to the procedure. At step 508, theannotations may be stored in the surgical annotations database 114. Atstep 510, the medical imaging data, annotations, and other data relatedto the VR practice session may be sent to the Augmented Reality (AR)assistance module 214.

FIG. 6 shows a flowchart 600 for a method executed by AR assistancemodule 214 in FIG. 2A. The AR assistance module 214 may support the userwhile performing the real surgical procedure on the subject patient. Atstep 602, the AR assistance module 214 may receive the medical imagingdata, annotations, and other data related to the VR practice session. Atstep 604, a position of the patient may be identified relative to an ARdisplay 118 worn by the user. Based on the identified position, at step606, user requested data may be overlaid on the portion.

In at least one non-limiting example, at step 608, the user may startperforming the surgical procedure on the patient, and the annotationsmay be presented to the user at designated times. By at least oneexample embodiment, time stamps may be used to determine an appropriatetime to present the annotations. For example, an annotation may be setto be presented five minutes into the surgical procedure.

Alternatively, an appropriate time to present the annotations may bedetermined based upon initiation of a particular step of a surgicalprocedure. For example, an annotation may be set to be presented duringinitiation of a third step of the surgical procedure. Each step of thesurgical procedure may be monitored by the AR display 118, and thus thesystem 102 may present the annotation to the user during a predeterminedstep.

The annotations may help the surgeon by storing important detailsrelated to any step of the surgical procedure, which may then berecalled during subsequent procedures. Such details may be presented tothe surgeon, at designated moments, for reminding about taking due care,as annotated. Thus, the surgeons may be assisted by their own personinput defined during a training session prior to the actual surgicalprocedure. This helps in improving accuracy of the surgeons by allowingthem to utilize every minute yet essential detail, thus reducingoccurrence of adverse events experienced by the patient, during asurgical procedure.

FIG. 7 shows a flowchart 700 for a method of providing surgicalassistance during a surgical procedure, according to an embodiment ofsystem 102 as shown in FIG. 2A.

In flowchart 700, each block may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing a respective specified logical function(s). In somealternative implementations, the functions noted in the blocks may occurin a different order than that shown in FIG. 7 . For example, two blocksshown in succession may in fact be executed substantially concurrentlyor the blocks may be executed in reverse order, depending upon thefunctionality involved. Any process descriptions or blocks in flowchartsshould be understood as representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or steps in the process, and alternateimplementations are included within the scope of the example embodimentsin which functions may be executed out of order from that shown ordiscussed, including substantially concurrently or in reverse order,depending on the functionality involved. In addition, the processdescriptions or blocks in flow charts should be understood asrepresenting decisions made by a hardware structure such as a statemachine. The flowchart 700 starts at the step 702 and proceeds to step712.

At step 702, a medical condition of a patient may be determined. Basedon the medical condition, surgical options may be determined for thepatient. In one embodiment, the surgical options may be determined byprocessor 202 of the system 102, as shown in FIG. 2A, in communicationwith the surgical options database 112.

At step 704, an optimal surgical option may be determined from among thestored surgical options based on outcomes of the surgical options inother patients. In one non-limiting example, the optimal surgical optionmay be determined by the processor 202.

At step 706, adverse events related to the surgical procedure may beidentified. In one non-limiting example, the adverse events may beidentified by the processor 202.

At step 708, a Three-Dimensional (3D) model of the patient may becreated using images of the patient gathered from one or more ofdifferent sources such as a digital camera, X-ray device, and MagneticResonance Imaging (MRI) device. In one non-limiting example, the 3Dmodel may be created by the processor 202.

At step 710, a Virtual Reality (VR) simulation may be created fortraining a surgeon using the 3D model. During the training, annotationsmay be received from the surgeon and stored in the surgical annotationsdatabase 114. The annotations may later be presented to the surgeon atdesignated moments during a real surgical procedure. The designatedmoments may be set based on time-stamps or steps of an actual surgicalprocedure. In one non-limiting example, the VR simulation may be createdand the annotations may be accepted by the processor 202.

At step 712, the annotations may be presented to the surgeon at thedesignated moments of an actual surgical procedure. The annotationsprovide surgical assistance to the surgeon by providing reminders ofcertain important activities and details to be implemented at thedesignated moments. In one non-limiting example, the annotations may bevisually presented by a user interface.

Functioning of the surgical planning module 1210, as shown in FIG. 2B,will now be explained with reference to flowchart 800 shown in FIG. 8 ,with reference to system 1102 of FIGS. 1B and 2B. The functionsperformed in the processes and methods may be implemented in differingorder than those presently depicted and described. Furthermore, theoutlined steps and operations are only provided as examples, and some ofthe steps and operations may be optional, combined into fewer steps andoperations, or expanded into additional steps and operations withoutdetracting from the essence of the disclosed embodiments.

At first, the surgeon may need to log-in to the surgical planning module1210. The surgeon may log-in using his credentials, i.e., a user nameand a password, using, for example, the user device 1116. Upon log-in,the surgical planning module 1210 may facilitate the surgeon identifyinga subject patient and receiving identification information for thesubject patient, at step 802. The surgical planning module 1210 may beutilized to retrieve medical data pertaining to the subject patient fromElectronic Health Records (EHR) stored in the real-time health recordunit 1108. The medical data may include medical images of an affectedbody part of the subject patient, as captured by the imaging system 1118using one or more medical imaging techniques, such as Magnetic ResonanceImaging (MRI), Computerized Tomography (CT), and X-Ray.

At step 804, a recommendation for surgery for the subject patient may bemade, and the diagnosis of the subject patient may be retrieved based atleast on the subject patient's medical data. At step 806, anyabnormalities present in the medical images may be identified; and atstep 808, the identified abnormalities may be correlated with thediagnosis of the subject patient.

At decision block 810, the surgical planning module 1210, shown in FIG.2B, may be utilized to determine whether the identified abnormalitiesand the subject patient's diagnosis are consistent with each other. Atstep 812 (“no”), if the identified abnormalities are not consistent withthe subject patient's diagnosis, the surgical planning module 1210 maybe utilized to alert the surgeon that a diagnosis is needed for thesubject patient. At step 814 (“yes”), In case the identifiedabnormalities are determined to be consistent with the subject patient'sdiagnosis, the surgical planning module 1210 may be utilized to segmentthe medical images. Thereafter, at step 816, the surgical planningmodule 1210 may be utilized to run the surgical annotation module 1212.

Functioning of the surgical annotation module 1212, as shown in FIG. 2B,will now be explained with reference to the flowchart 900 shown in FIG.9 , with reference to system 1102 of FIGS. 1B and 2B. Once again, thefunctions performed in the processes and methods may be implemented indiffering order. Furthermore, the outlined steps and operations are onlyprovided as examples, and some of the steps and operations may beoptional, combined into fewer steps and operations, or expanded intoadditional steps and operations without detracting from the essence ofthe disclosed embodiments.

At step 902, the surgical annotation module 1212 may receiveidentification information for the subject patient, the recommendedsurgery for the subject patient, the subject patient's diagnosis, andthe image segments, from the surgical planning module 1210, shown inFIG. 2B. At step 904, the surgical annotation module 1212 may beutilized to display the image segments for the surgeon. The imagesegments may be displayed on the GUI of the user device 1116. Forexample, as shown in FIG. 10A, segment of an affected body part, i.e., aruptured Achilles tendon of a right ankle of the subject patient, isdisplayed to the surgeon.

At step 906, the surgical paths may be retrieved from the surgical pathdatabase 1114 based on criteria set by the surgeon, including, but notlimited to, frequency of success of the surgical paths, time ofoperation for the surgical paths, etc. Thereafter, the surgical pathsmay be overlaid on the image segments of the affected body part of thesubject patient.

At step 908, the surgical paths that are overlaid on the image segmentsmay be displayed for the surgeon using the GUI of the user device 1116.For example, as shown in FIG. 10B, the surgical paths overlaid on animage segment of the affected body part, i.e., the ruptured Achillestendon of the right ankle of the subject patient, is displayed indifferent colors, such as red, green, yellow, etc.

At step 910, a surgical path may be selected from the surgical paths,and a surgical step belonging to the surgical path may be selected basedon the surgeon's input. For example, as shown in FIG. 10B, the surgicalpaths may be displayed to the surgeon for selecting a surgical path,i.e., a route. Further, as shown in FIG. 10C, the surgical steps may bedisplayed to the surgeon to select a surgical step in order to repairthe right knee of the subject patient.

In one non-limiting example, the surgeon may select the surgical stepbased at least on criteria, including but not limited to average time ofthe surgical step and worst-case risks of the surgical step. In onenon-limiting example, as shown in FIG. 10D, the surgical annotationmodule 1212 may be utilized to display the selected surgical step, i.e.,surgical step 4 for the surgeon on the GUI of the user device 1116.

At step 912, an image segment may be selected from the image segments,based on the surgeon's input. For example, as shown in FIG. 10D, thesurgical annotation module 1212 may be utilized to enable the surgeon toselect an image segment using the user device 1116. In a non-limitingexample, the surgical annotation module 1212 may be utilized to enablethe surgeon to include or exclude the image segments. For example, asshown in FIG. 10E, the surgical annotation module 1212 may be utilizedto enable the surgeon to include or exclude the image segments relatedto the right ankle.

At step 914, annotations related to the surgical step may be receivedfrom the surgeon via, e.g., the GUI of the user device 1116, which mayprovide input options for the user. For example, as shown in FIG. 10F,the surgeon may provide an annotation, such as “to move a tibial nervemedially at the surgical step 4 to avoid contact with the tibial nerve”in order to prevent damage to the tibial nerve during execution ofsurgical step 4.

In another example, the surgeon may want to add an annotation at aparticular surgical step regarding a particular concern pertaining to avascular tissue in the right knee. In that example scenario, the surgeonmay want to include the vascular tissue and nerve tissue but excludespecific tendons that may obscure a view of the vascular tissue and thenerve tissue. Other examples of the annotations may include requesting a30% reduction in volume at a particular surgical step, requesting asurgical implant being added at the particular surgical step, etc.

At step 916, the image segment along with the annotations may be storedin the surgical annotation database 1110.

FIG. 11 a flowchart 11100 showing a method for creating the surgicalplan for use in the surgical procedure, according to an embodiment,explained in conjunction with the elements disclosed in FIGS. 1B, 2B,and 10A-10F.

In flowchart 11100, each block may represent a module, segment, orportion of code, which includes one or more executable instructions forimplementing a respective specified logical function(s). In somealternative implementations, the functions noted in the blocks may occurin a different order than that shown in FIG. 11 . For example, twoblocks shown in succession may in fact be executed substantiallyconcurrently or the blocks may be executed in the reverse order,depending upon the functionality involved. Any process descriptions orblocks in flowcharts should be understood as representing modules,segments, or portions of code which include one or more executableinstructions for implementing specific logical functions or steps in theprocess, and alternate implementations are included within the scope ofthe example embodiments in which functions may be executed out of orderfrom that shown or discussed, including substantially concurrently or inreverse order, depending on the functionality involved. In addition, theprocess descriptions or blocks in flow charts should be understood asrepresenting decisions made by a hardware structure, such as a statemachine. The flowchart 11100 starts at step 11102 and ends at step11116.

At step 11102, a recommended surgery for a subject patient may beidentified based at least on medical data of the subject patient, whichmay include one or more medical images of an affected body part of thesubject patient.

At step 11104, one or more abnormalities present in the one or moremedical images of the subject patient may be identified.

At step 11106, the one or more medical images may be segmented into oneor more image segments.

At step 11108, surgical paths may be retrieved for addressing thesurgical need of the subject patient from the surgical path database1114, shown in FIG. 1B.

At step 11110, the surgical paths overlaid on the one or more imagesegments may be displayed for the surgeon on the user device 1116.

At step 11112, a surgical path may be selected from the surgical paths,a surgical step belonging to the surgical path may be selected, and animage segment from may be selected the one or more image segments, basedon the surgeon's input.

At step 11114, one or more annotations related to the surgical step maybe received from the surgeon.

At step 11116, the image segment may be stored along with the one ormore annotations, in the surgical annotation database 1110, therebycreating the surgical plan for use in the surgical procedure.Thereafter, the surgical plan may be displayed on an operating roomdisplay 1120 during the surgical procedure.

In an illustrative embodiment, any of the operations, processes, etc.,described herein may be implemented as computer-readable instructionsstored on a computer-readable medium. The computer-readable instructionscan be executed by a processor of a mobile unit, a network element,and/or any other computing device.

There is little distinction left between hardware and softwareimplementations of aspects of systems; the use of hardware or softwareis generally (but not always, in that in certain contexts the choicebetween hardware and software can become significant) a design choicerepresenting cost vs. efficiency tradeoffs. There are various vehiclesby which processes and/or systems and/or other technologies describedherein can be effected, e.g., hardware, software, and/or firmware, andthat the preferred vehicle will vary with the context in which theprocesses and/or systems and/or other technologies are deployed. Forexample, if an implementer determines that speed and accuracy areparamount, the implementer may opt for a mainly hardware and/or firmwarevehicle; if flexibility is paramount, the implementer may opt for amainly software implementation; or, yet again alternatively, theimplementer may opt for some combination of hardware, software, and/orfirmware.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment,several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in integratedcircuits, as one or more computer programs running on one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs running on one or more processors(e.g., as one or more programs running on one or more microprocessors),as firmware, or as virtually any combination thereof, and that designingthe circuitry and/or writing the code for the software and or firmwarewould be well within the skill of one of skill in the art in light ofthis disclosure. In addition, those skilled in the art will appreciatethat the mechanisms of the subject matter described herein are capableof being distributed as a program product in a variety of forms, andthat an illustrative embodiment of the subject matter described hereinapplies regardless of the particular type of signal bearing medium usedto actually carry out the distribution. Examples of a signal bearingmedium include, but are not limited to, the following: a recordable typemedium such as a floppy disk, a hard disk drive, a CD, a DVD, a digitaltape, a computer memory, etc.; and a transmission type medium such as adigital and/or an analog communication medium (e.g., a fiber opticcable, a waveguide, a wired communications link, a wirelesscommunication link, etc.).

The embodiments, features, systems, devices, materials, methods andtechniques described herein may, in some embodiments, be similar to anyone or more of the embodiments, features, systems, devices, materials,methods and techniques described in U.S. application Ser. No.16/012,464, filed Jun. 19, 2018, titled SURGERY PLANNING, which isincorporated by reference in its entirety. In addition, the embodiments,features, systems, devices, materials, methods, and techniques describedherein may, in certain embodiments, be applied to or used in connectionwith any one or more of the embodiments, features, systems, devices, orother matter. All of the patents and applications are incorporated byreference in their entireties.

Those skilled in the art will recognize that it is common within the artto describe devices and/or processes in the fashion set forth herein,and thereafter use engineering practices to integrate such describeddevices and/or processes into data processing systems. That is, at leasta portion of the devices and/or processes described herein can beintegrated into a data processing system via a reasonable amount ofexperimentation. Those having skill in the art will recognize that atypical data processing system generally includes one or more of asystem unit housing, a video display device, a memory such as volatileand non-volatile memory, processors such as microprocessors and digitalsignal processors, computational entities such as operating systems,drivers, graphical user interfaces, and applications programs, one ormore interaction devices, such as a touch pad or screen, and/or controlsystems including feedback loops and control motors (e.g., feedback forsensing position and/or velocity; control motors for moving and/oradjusting components and/or quantities). A typical data processingsystem may be implemented utilizing any suitable commercially availablecomponents, such as those typically found in datacomputing/communication and/or network computing/communication systems.

The herein described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely examples, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected”, or“operably coupled”, to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable”, to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

From the foregoing, it will be appreciated that various embodiments ofthe present disclosure have been described herein for purposes ofillustration, and that various modifications may be made withoutdeparting from the scope and spirit of the present disclosure.Accordingly, the various embodiments disclosed herein are not intendedto be limiting.

We claim:
 1. A computer-implemented method for presenting digitalsimulation annotations to a surgeon for an image-guided surgicalprocedure, the method comprising: identifying at least one surgicalprocedure to be performed on an organ of a patient having an identifiedcondition, wherein the at least one surgical procedure is identifiedbased on at least one prior procedure performed on one or more otherpatients with the identified condition: retrieving comparative data ofthe one or more other patients; determining a set of comparativepatients from the one or more patients based on one or more similaritiesbetween the retrieved comparative data and data of the patient;determining attributes for the set of comparative patients; determiningone or more correlation coefficients for at least one adverse event thatmay occur during the at least one surgical procedure; determining aplurality of surgical paths to the organ to address the identifiedcondition; for each of the plurality of surgical paths, determining anadverse event score based on a relationship between the correlationcoefficients and the attributes; selecting a surgical path for treatingthe identified condition to perform on the organ of the patient by asurgeon based on a ranking of the adverse event scores; producing adigital simulation of the selected surgical path to the organ for thepatient for displaying data associated with at least one adverse event,wherein the digital simulation includes at least one of a virtualreality (VR) simulation or an augmented reality (AR) simulation;annotating the digital simulation with timestamped inputted annotationsfor avoiding the at least one adverse event; and displaying, via anelectronic display viewable by the surgeon, images for an image-guidedsurgical procedure based on the selected surgical path and thetimestamped inputted annotations to assist the surgeon for avoiding theat least one adverse event during the image-guided surgical procedure.2. The method of claim 1, further comprising performing the digitalsimulation using a surgical practice equipment configured to allow for avirtual simulation of a surgical procedure, wherein the surgicalpractice equipment includes: a simulation display configured to displaythe organ be operated upon, and a virtual force apparatus including aphysical manipulation device configured to be manipulated by a user toperform one or more virtually surgical steps on the displayed organ, anda tracking device configured to track motion of the physicalmanipulation device.
 3. The method of claim 1, wherein determining theset of comparative patients includes: generating a three-dimensional(3D) model of a region of the patient; generating comparative threedimensional (3D) models based on the retrieved comparative data of theone or more other patients; and identifying the one or more similaritiesbetween the 3D model and the comparative 3D models.
 4. The method ofclaim 1, wherein determining the set of comparative patients includes:comparing virtual dimensional (3D) model of region of the patient andcomparative virtual dimensional (3D) models of the one or more otherpatients; and selecting the comparative other patients based on thecomparison.
 5. The method of claim 1, wherein the timestamped inputtedannotations include at least one of audio annotations, videoannotations, or text annotations.
 6. The method of claim 1, wherein thedigital simulation is performed using equipment configured to simulatingat least one of moving one or more instruments, cutting, suturing, acoagulating step, or an organ-specific surgical step.
 7. The method ofclaim 1, further comprising: receiving annotations via digitalsimulation equipment configured to display the digital simulation; andapplying time stamps to user-inputted annotations correlating theannotations to the digital simulations, thereby producing thetimestamped inputted annotations.
 8. The method of claim 1, wherein thedigital simulation includes virtually simulating one or more surgicalsteps performed on a virtual model of the organ.
 9. The method of claim1, further comprising: receiving surgeon input; and modifying thetimestamped inputted annotations based on the received surgeon input,wherein the displayed timestamped inputted annotations include themodified time-stamped annotations.
 10. The method of claim 1, whereinthe at least one adverse event includes at least one of a blood loss ora drop in blood pressure.
 11. A computer-implemented method forpresenting digital simulation annotations to a surgeon for animage-guided surgical procedure, the method comprising: retrievingcomparative data of one or more reference patients based on anidentified condition of a patient; determining a set of comparativereference patients from the one or more reference patients based on oneor more similarities between the retrieved comparative data and data ofthe patient; determining, using one or more virtual simulations usingthree-dimensional virtual models of the organ of the patient, aplurality of surgical paths to the organ to address the identifiedcondition; for each of the plurality of surgical paths, determining anadverse event score based on the set of comparative reference patients;selecting one of the surgical paths to the organ of the patient based ona ranking of the adverse event scores; producing a digital simulation ofthe selected surgical path to the organ for displaying data associatedwith at least one of the adverse events, wherein the digital simulationincludes at least one of a virtual reality (VR) simulation or anaugmented reality (AR) simulation; annotating the digital simulationwith timestamped annotations for avoiding the at least one adverseevent; displaying images for an image-guided surgical procedure based onthe selected surgical path to assist with treatment of the patient'sorgan having the identified condition; and intra-operatively presentingthe timestamped annotations to avoid the at least one adverse eventduring the image-guided surgical procedure.
 12. The method of claim 11,wherein determining the set of comparative patients includes: generatinga three-dimensional (3D) model of the organ; generating comparativethree dimensional (3D) models based on the retrieved comparative data ofthe one or more reference patients; and identifying the one or moresimilarities between the 3D model and the comparative 3D models.
 13. Themethod of claim 11, wherein determining the set of comparative patientsincludes: comparing virtual dimensional (3D) model of region of thepatient and comparative virtual dimensional (3D) models of the one ormore reference patients; and selecting the comparative other patientsbased on the comparison.
 14. The method of claim 11, wherein thetimestamped annotations comprise at least one of audio annotations,video annotations, or text annotations.
 15. A system for presentingdigital simulation annotations to a surgeon for an image-guided surgicalprocedure, the system comprising: one or more processors; and one ormore memories storing instructions that, when executed by the one ormore processors, cause the system to perform a process comprising:retrieving comparative data of one or more reference patients based onan identified condition of a patient; determining a set of comparativereference patients from the one or more reference patients based on oneor more similarities between the retrieved comparative data and data ofthe patient; determining, using one or more virtual simulations usingthree-dimensional virtual models of the organ of the patient, aplurality of surgical paths to the organ to address the identifiedcondition; for each of the plurality of surgical paths, determining anadverse event score based on the set of comparative reference patients;selecting one of the surgical paths to the organ of the patient based ona ranking of the adverse event scores; producing a digital simulation ofthe selected surgical path to the organ for displaying data associatedwith at least one of the adverse events, wherein the digital simulationincludes at least one of a virtual reality (VR) simulation or anaugmented reality (AR) simulation; annotating the digital simulationwith timestamped annotations for avoiding the at least one adverseevent; displaying images for an image-guided surgical procedure based onthe selected surgical path to assist with treatment of the patient'sorgan having the identified condition; and intra-operatively presentingthe timestamped annotations to avoid the at least one adverse eventduring the image-guided surgical procedure.
 16. The method of claim 15,wherein the process further comprises: generating a three-dimensional(3D) model of the organ; generating comparative three dimensional (3D)models based on the retrieved comparative data of the one or morereference patients; and identifying the one or more similarities betweenthe 3D model and the comparative 3D models.
 17. The method of claim 15,wherein determining the set of comparative patients includes: comparingvirtual dimensional (3D) model of region of the patient and comparativevirtual dimensional (3D) models of the one or more reference patients;and selecting the comparative other patients based on the comparison.